| Name of the Table | Source |
|---|---|
| D1_1_SDG | dashboards.sdgindex.org |
| D2_2_Unemployment_rate | ilo.org |
| D3_0_GDP_per_capita | data.worldbank.org |
| D3_1_Military_expenditure_percent_GDP | data.worldbank.org |
| D3_2_Military_expenditure_percent_gov_exp | data.worldbank.org |
| D4_0_Internet_usage | ourworldindata.org |
| D5_0_Human_freedom_index | cato.org |
| D6_0_Disaters | kaggle.com |
| D7_0_COVID | github.com |
| D8_0_Conflicts | datacatalog.worldbank.org |
Data
Sources
We are collecting our Data from the sustainability development report (SDG), the international labour organization (ILOSTAT), the World Bank, Our world in data, the CATO institute, one from Kaggle (disasters: we couldn’t find relevant accessible information from somewhere else) and GitHub. We found different datasets containing useful information in relation with the SDGs. The details about these data and the links are presented in the next section. Utilizing the kableExtra package, we provide a comprehensive list and corresponding links to our sources, as outlined below:
Description
During the wrangling process, we added data to our table (D1_1_SDG) from different other datasets and match them based on the country code, and the year. The tables below show all the variables present in our 9 databases. We will then merge them to have our final table for the analysis.
Our databases
Sustainable Development Goals database (D1_1_SDG)
The Sustainable Development Goals (SDGs) are a universal set of 17 interlinked goals that were adopted by the United Nations in 2015 as part of the 2030 Agenda for Sustainable Development. These goals provide a shared blueprint for peace and prosperity for people and the planet, now and into the future.
Our primary database focuses on the Sustainable Development Goals (SDG). Below is a table summarizing the key variables included:
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2022) |
| overallscore | Overall score on all 17 SDGs (the score are % of achievement of the goals determined by the UN based on several indicators) |
| goal1:goal17 | Score on each SDG except SDG 14 (16 variables) |
| population | Population of the country |
Unemployment rate database (D2_2_Unemployment_rate)
This database give us comprehensive data on the unemployment rates for each country from 2000 to 2022. Originally, it included categories based on various age groups. However, for simplicity and coherence, the database has been streamlined to focus exclusively on the unemployment rates of individuals aged 15 years and older.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2022) |
| unemployment.rate | Unemployment rate (% of the population 15 years old and older) |
GDP per capita database (D3_0_GDP_per_capita)
This database offers detailed information on the GDP per capita in dollars for various countries, covering the period from 2000 to 2022. It is designed to provide insights into the economic performance of each country over these years, measured through the lens of per capita GDP.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2022) |
| GDPpercapita | GDP per capita |
Proportion of the GDP dedicated to Military expenditures database (D3_1_Military_expenditure_percent_GDP)
This database provides an insightful view of the proportion of their respective GDPs that countries have allocated to military expenditures. It covers the period from 2000 to 2022.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2022) |
| MilitaryExpenditurePercentGDP | Military expenditures in percentage of GDP |
Internet usage database (D4_0_Internet_usage)
This database provides information on the percentage of the population that uses the internet in each country. It covers the period from 2000 to 2022.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2022) |
| internet.usage | Internet usage (% of the population) |
Human freedom index database (D5_0_Human_freedom_index)
This database provides information on the Human Freedom Index (HFI) for each country. The HFI is a composite index that measures the degree to which people are free to enjoy important rights and freedoms. It covers the period from 2000 to 2022.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2022) |
| region | Part of the world, group of countries (e.g. Eastern Europe, Dub-Saharan Africa, South Asia, etc.) |
| pf_law | Rule of law, mean score of: Procedural justice, Civil, justice, Criminal justice, Rule of law (V-Dem) |
| pf_security | Security and safety, mean score of: Homicide, Disappearances conflicts, terrorism |
| pf_movement | Freedom of movement (V-Dem), Freedom of movement (CLD) |
| pf_religion | Freedom of religion, Religious organization, repression |
| pf_assembly | Civil society entry and exit, Freedom of assembly, Freedom to form/run political parties, Civil society repression |
| pf_expression | Direct attacks on the press, Media and expression (V-Dem), Media and expression (Freedom House), Media and expression (BTI), Media and expression (CLD) |
| pf_identity | Same-sex relationships, Divorce, Inheritance rights, Female genital mutilation |
| ef_gouvernment | Government consumption, Transfers and subsidies, Government investment, Top marginal tax rate, State ownership of assets |
| ef_legal | Judicial independence, Impartial courts, Protection of property rights, Military interference Integrity of the legal system Legal enforcementof contracts, Regulatory costs, Reliability of police |
| ef_money | Money growth, Standard deviation of inflation, Inflation: Most recent year, Freedom to own foreign currency |
| ef_trade | Tariffs, Regulatory trade barriers, Black-market exchange rates, Movement of capital and people |
| ef_regulation | Credit market regulations, Labor market regulations, Business regulations |
Disaster list database (D6_0_Disaters)
This database provides information on the number of deaths, injured, affected and homeless people, as well as the total number of affected people and the total of infrastructure damages caused by disasters in each country. It covers the period from 2000 to 2021.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2021) |
| continent | Continents touched by the disasters such as floods, ouragan |
| total_deaths | Number of deaths caused by disasters |
| no_injured | Number of injured caused by disasters |
| no_affected | Number of affected caused by disasters |
| no_homeless | Number of homeless caused by disasters |
| total_affected | Total number of affected caused by disasters |
| total_damages | Total of infrastructure damages |
COVID database (D7_0_COVID)
This database provides information on the number of people dead due to COVID, the number of COVID cases and the Government Response Stringency Index in each country. It covers the period from 2020 to 2022.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2020-2022) |
| deaths_per_million | Number of people dead due to COVID |
| cases_per_million | Number of COVID cases |
| stringency | Government Response Stringency Index: composite measure based on 9 response indicators including school closures, workplace closures, and trave |
Conflicts database (D8_0_Conflicts)
This database provides information on the number of deaths, the number of people affected and the maximum intensity of conflicts in each country. It covers the period from 2000 to 2022.
| Variable Name | Explanation |
|---|---|
| code | Country code (ISO) |
| country | Country name |
| year | Year of the observation (2000-2022) |
| ongoing | Variable coded 1 for more than 25 deaths in intrastate conflict and 0 otherwise according to UCDP/PRIO Armed Conflict Dataset 17.1. |
| sum_deaths | Best estimate of deaths in all categories of violence (non-state, one-sided and state-based) recorded by the Uppsala Conflict Data Program in the country based on the UCDP GED dataset (unpublished 2016 data). The location of these events is used for estimating the extent of violence. |
| pop_affected | Share of population affected by violence in percentage (0 to 100) measured as described above based on population data from CIESIN, the PRIO-GRID structure as well as UCDP GED. |
| area_affected | Area affected by conflict |
| maxintensity | Two different intensity levels are coded: minor armed conflicts (1) and wars (2), Takes the max intensity of conflict in the country so that it is coded 2 if there is at least one war (>=1000 deaths in intrastate conflict) during the year. Data from UCDP/PRIO Armed Conflict Dataset 17.1. |
Wrangling/cleaning
Pre-cleaning
To deal with the large scale of the datasets, we pre-cleaned each dataset before merging them together. This streamlined the process, simplifying the cleaning of the final, combined dataset. The treatment of missing values will be taken care of after merging our datasets.
Dataset on SDG
This is our main dataset that we clean in order to keep the columns containing the following information: country name, country code, year, population, overall score and the SDGs scores.
We start by importing the data and converting it into a DataFrame. Next, we rename the columns and convert the scores into numeric variables.
#### D1_0_SDG importation ####
#Importing the data from the csv file
D1_0_SDG <- read.csv(here("scripts","data","SDG.csv"), sep = ";")
# Keeping only the firste 22 columns
D1_0_SDG <- D1_0_SDG[,1:22]
# Renaming the columns
colnames(D1_0_SDG) <- c("code", "country", "year", "population",
"overallscore", "goal1", "goal2", "goal3",
"goal4", "goal5", "goal6", "goal7", "goal8",
"goal9", "goal10", "goal11", "goal12",
"goal13", "goal14", "goal15", "goal16",
"goal17")
# Function to convert the overallscore column into numeric values
# and replace the "," by "."
D1_0_SDG[["overallscore"]] <-
as.double(gsub(",", ".", D1_0_SDG[["overallscore"]]))
# Function to convert the scores columns into numeric values
# and also to replace the "," by "."
makenumSDG <- function(D1_0_SDG) {
for (i in 1:17) {
varname <- paste("goal", i, sep = "")
D1_0_SDG[[varname]] <-
as.double(gsub(",", ".", D1_0_SDG[[varname]]))
}
return(D1_0_SDG)
}
# Applying the function to the 1_0_SDG dataset
D1_0_SDG <- makenumSDG(D1_0_SDG)We proceed by examining the missing values.
#### D1_0_SDG missing values ####
propmissing <- numeric(length(D1_0_SDG))
for (i in 1:length(D1_0_SDG)){
proportion <- mean(is.na(D1_0_SDG[[i]]))
propmissing[i] <- proportion
}
variable_names <- colnames(D1_0_SDG)
prop_missing_data <- data.frame(variable = variable_names, prop_missing = propmissing)
ggplot(prop_missing_data,
aes(x = variable,
y = prop_missing)) +
geom_bar(stat = "identity",
aes(fill = prop_missing),
color = "black") +
scale_fill_viridis_c(name = "Proportion\nof missing",
begin = 0.5) +
labs(title = "NAs by columns in the main dataset",
x = "Variable",
y = "Proportion of Missing Values") +
theme(plot.title = element_text(size = 10, hjust = 0.5),
axis.title.x = element_text(size = 8),
axis.title.y = element_text(size = 8)) +
theme_minimal() +
coord_flip() +
guides(fill = FALSE)
Observing that the ‘population’ column contains numerous NAs, we investigate and discover that missing values are common, as some observations represent regions, not countries. Therefore, we can safely exclude these observations.
#### D1_0_SDG missing values in population ####
SDG0 <- D1_0_SDG %>%
group_by(code) %>%
select(population) %>%
summarize(NaPop = mean(is.na(population))) %>%
filter(NaPop != 0)
ggplot(SDG0,
aes(x = code,
y = NaPop)) +
geom_bar(stat = "identity",
fill = viridis(1,
begin = 0.7),
color = "black") +
labs(title = "NAs by row in population variable are for regions and not countries",
x = "Code",
y = "Proportion of NAs") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45,
hjust = 1),
plot.title = element_text(hjust = 0.5))
D1_0_SDG <- D1_0_SDG %>%
filter(!str_detect(code, "^_"))
Now, there are no missing values in the ‘population’ variable, and we observe that it contains information on 166 countries.
We notice that NAs are present in only three SDG scores: 1, 10, and 14. Additionally, when a country has NAs, they occur across all years or not at all. Consequently, we decide to conduct further investigations on these three SDG scores to determine whether to include them in our analysis.
For goal 1, there are only 9.04% missing values in 15 different countries. Goal 1 being “End poverty”, we decide to keep it and only remove the countries with no information for the analysis.
#### SDG2 missing values ####
SDG2 <- D1_0_SDG |>
group_by(code) |>
select(contains("goal")) |>
summarize(Na1 = mean(is.na(goal1))) |>
filter(Na1 != 0)
country_number <- length(unique(D1_0_SDG$country))
length(unique(SDG2$code))/country_number
#> [1] 0.0904For goal 10, there are only 10.2% missing values in 17 different countries. Goal 10 being “reduced inequalities”, we decide to keep it and only remove the countries with no information for the analysis.
#### SDG3 missing values ####
SDG3 <- D1_0_SDG |>
group_by(code) |>
select(contains("goal")) |>
summarize(Na10 = mean(is.na(goal10))) |>
filter(Na10 != 0)
length(unique(SDG3$code))/country_number
#> [1] 0.102For goal 14, there are 24.1% missing values in 40 different countries. Goal 14 being “life under water”, we decide not to keep it, because other SDG such as “life on earth” and “clean water” already treat similar subjects.
#### SDG4 missing values ####
SDG4 <- D1_0_SDG |>
group_by(code) |>
select(contains("goal")) |>
summarize(Na14 = mean(is.na(goal14))) |>
filter(Na14 != 0)
length(unique(SDG4$code))/country_number
#> [1] 0.241
D1_0_SDG <- D1_0_SDG %>%
select(-goal14)We will work with various datasets and merge them using the country code and year as key identifiers. To ensure accurate matching, we first verify that country names are encoded in UTF-8 format. Then, we standardize the names of the countries (requiring a custom match for Turkey) and the country codes, utilizing the countrycode library. Additionally, we compile a list of all country codes from the main database to filter the other datasets. Lastly, we complete the database to include all possible “country, year” combinations, ensuring the total number of rows remains unchanged.
#### D1_0_SDG country code ####
D1_0_SDG$country <- stri_encode(D1_0_SDG$country, to = "UTF-8")
D1_0_SDG <- D1_0_SDG %>%
mutate(country = countrycode(country, "country.name", "country.name",
custom_match = c("T�rkiye"="Turkey")))
D1_0_SDG$code <- countrycode(
sourcevar = D1_0_SDG$code,
origin = "iso3c",
destination = "iso3c",
)
list_country <- c(unique(D1_0_SDG$code))
D1_0_SDG_country_list <- D1_0_SDG %>%
filter(code %in% list_country) %>%
select(code, country)
D1_0_SDG_country_list <- D1_0_SDG_country_list %>%
select(code, country) %>%
distinct()Finally, we complete the database to ensure there are no missing pairs of (year, code).
Here are the first few lines of the cleaned dataset on SDG achievement scores:
For this first dataset, we reduced the size from 4,140 observations across 120 variables to 3,818 observations for 21 variables.
As said, this is now our main dataset. All subsequent datasets will be merged with this dataset. Therefore, for all the following datasets, we will make sure that we only keep data for the same countries and years as in this dataset. We have a total of 166 countries and the years range from 2000 to 2022.
Dataset on Unemployment rate
In this dataset, the initial step involves importing the data. Next, we ensure that the names and codes of the countries are formatted in UTF-8, preventing any discrepancies due to mismatches in country names. Following this, we modify the column names and filter the data to include only the relevant countries and years, specifically the years 2000 to 2022, covering 166 countries from our primary dataset.
#### D2_1_Unemployment_rate pre-cleaning ####
D2_1_Unemployment_rate <-
read.csv(here("scripts","data","UnemploymentRate.csv")) %>%
mutate(country = iconv(ref_area.label,
to = "UTF-8",
sub = "byte"),
country = countrycode(country,
"country.name",
"country.name"),
`unemployment rate` = obs_value / 100,
year = time,
age_category = classif1.label,
sex = sex.label) %>%
select(-ref_area.label, -time, -obs_value,
-classif1.label, -sex.label, -source.label,
-obs_status.label, -indicator.label) %>%
merge(D1_0_SDG_country_list[, c("country", "code")],
by = "country",
all.x = TRUE) %>%
filter(year >= 2000 & year <= 2022,
!str_detect(sex, fixed("Male")) & !str_detect(sex, fixed("Female")),
code %in% D1_0_SDG_country_list$code,
age_category == "Age (Youth, adults): 15+") %>%
select(code,
country,
year,
`unemployment rate`) %>%
distinct()Here are the first few lines of the cleaned dataset on Unemployment rate:
For this first dataset, we reduced the size from 82,800 observations across 8 variables to 3,812 observations for 5 variables.
Dataset on GDP military Expenditures
We have three different databases which contain information on each countries over the years. Each year represent one variable. We want to extract three variables for our analysis: GDP per capita, military expenditures in percentage of the GDP and military expenditures in percentage of government expenditures.
#### GDP per capita pre-cleaning ####
GDPpercapita <-
read.csv(here("scripts","data","GDPpercapita.csv"),
sep = ";")
MilitaryExpenditurePercentGDP <-
read.csv(here("scripts","data","MilitaryExpenditurePercentGDP.csv"),
sep = ";")
MiliratyExpenditurePercentGovExp <-
read.csv(here("scripts","data","MiliratyExpenditurePercentGovExp.csv"),
sep = ";")After importing the data, we fill in the missing country codes using the column Indicator.Name, because we realized after some manipulations, that some of the country codes were false, but the next column contained the right ones.
#### GDP per capita fill code ####
fill_code <- function(data){
data <- data %>%
mutate(Country.Code = ifelse(!grepl("^[A-Z]{3}$", Country.Code),
Indicator.Name, Country.Code))
}We create a set of functions that we will apply to each database. First, remove the variables that we don’t need, which are the years before 2000. Second, make sure that the values are numeric and rename the year variables (because they all had an “X” before year number). Third, transform the database from wide to long, in order to match the main database. Fourth, transform the year variable into an integer variable and rearrange and rename the columns to match the ones of the other databases. Then, we apply these transformations to the three databases.
#### Useful functions ####
remove <- function(data){
years <- seq(1960, 1999)
removeyears <- paste("X", years, sep = "")
data <- data[, !(names(data) %in% c("Indicator.Name",
"Indicator.Code",
"X",
removeyears))]
}
makenum <- function(data) {
for (i in 2000:2022) {
year <- paste("X", i, sep = "")
data[[year]] <- as.numeric(data[[year]])
}
return(data)
}
renameyear <- function(data) {
for (i in 2000:2022) {
varname <- paste("X", i, sep = "")
names(data)[names(data) == varname] <- gsub("X", "", varname)
}
return(data)
}
wide2long <- function(data) {
data <- pivot_longer(data,
cols = -c("Country.Name",
"Country.Code"),
names_to = "year",
values_to = "data")
return(data)
}
yearint <- function(data) {
data$year <- as.integer(data$year)
return(data)
}
nameorder <- function(data) {
colnames(data) <- c("country",
"code",
"year",
"data")
data <- data %>% select(c("code",
"country",
"year",
"data"))
}
cleanwide2long <- function(data){
data <- fill_code(data)
data <- remove(data)
data <- makenum(data)
data <- renameyear(data)
data <- wide2long(data)
data <- yearint(data)
data <- nameorder(data)
}
GDPpercapita <-
cleanwide2long(GDPpercapita)
MilitaryExpenditurePercentGDP <-
cleanwide2long(MilitaryExpenditurePercentGDP)
MiliratyExpenditurePercentGovExp <-
cleanwide2long(MiliratyExpenditurePercentGovExp)We rename the colums with the main information, standardize the country code and remove the countries that are not in our main database. We see that all the 166 countries are there.
#### GDP per capita renamed and standardized ####
GDPpercapita <- GDPpercapita %>%
rename(GDPpercapita = data)
MilitaryExpenditurePercentGDP <- MilitaryExpenditurePercentGDP %>%
rename(MilitaryExpenditurePercentGDP = data)
MiliratyExpenditurePercentGovExp <- MiliratyExpenditurePercentGovExp %>%
rename(MiliratyExpenditurePercentGovExp = data)
GDPpercapita$code <- countrycode(
sourcevar = GDPpercapita$code,
origin = "iso3c",
destination = "iso3c",
)
MilitaryExpenditurePercentGDP$code <- countrycode(
sourcevar = MilitaryExpenditurePercentGDP$code,
origin = "iso3c",
destination = "iso3c",
)
MiliratyExpenditurePercentGovExp$code <- countrycode(
sourcevar = MiliratyExpenditurePercentGovExp$code,
origin = "iso3c",
destination = "iso3c",
)
GDPpercapita <- GDPpercapita %>%
filter(code %in% list_country)
length(unique(GDPpercapita$code))
#> [1] 166
MilitaryExpenditurePercentGDP <- MilitaryExpenditurePercentGDP %>%
filter(code %in% list_country)
length(unique(MilitaryExpenditurePercentGDP$code))
#> [1] 166
MiliratyExpenditurePercentGovExp <- MiliratyExpenditurePercentGovExp %>%
filter(code %in% list_country)
length(unique(MiliratyExpenditurePercentGovExp$code))
#> [1] 166There were only 157 countries that were both in the main SDG dataset and in these 3 datasets, but we suspected that some of the missing countries were in the database but not rightly matched. Indeed, Bahamas was in the database but instead of the code “BHS” there was “The”, for “COD” it was “Dem. Rep.”, for “COG” it was “Rep”, etc. We remarked that the code is in another column of the initial database: “Indicator.Name”. We went back to the initial database and before cleaning it we put the right codes (as seen above) and after rerunning the code we see that we have all our 166 countries from the initial dataset.
#### Missing countries ####
list_country_GDP <- c(unique(GDPpercapita$code))
setdiff(list_country, list_country_GDP)
#> character(0)#### Pre-cleaned datasets on GDP per capita ####
D3_1_GDP_per_capita <- GDPpercapita
D3_2_Military_Expenditure_Percent_GDP <- MilitaryExpenditurePercentGDP
D3_3_Miliraty_Expenditure_Percent_Gov_Exp <- MiliratyExpenditurePercentGovExpHere are the first few lines of the cleaned dataset of GDP per capita:
For this dataset, we went from ??? observations for 68 variables to 3818 observations for 4 varibles.
Here are the first few lines of the cleaned dataset of military expenditures in percentage of GDP:
For this dataset, we went from ??? observations for 68 variables to 3818 observations for 4 varibles.
Here are the first few lines of the cleaned dataset of military expenditures in percentage of government expenditures:
Dataset on internet usage
To prepare the dataset on internet usage in the world to be merge with the other data, we first, import the data. Then, we keep only the year that we are interested in (2000 to 2022). We also rename the column and keep only the country that match the list of the countries in the main dataset on the SDG.
#### Internet usage pre-cleaning ####
D4_0_Internet_usage <- read.csv(here("scripts", "data", "InternetUsage.csv")) %>%
filter(Year >= 2000, Year <= 2022) %>%
rename(
code = Code,
country = Entity,
year = Year,
internet_usage = Individuals.using.the.Internet....of.population.
) %>%
mutate(internet_usage = internet_usage / 100) %>%
filter(code %in% list_country) %>%
select(code, country, year, internet_usage)Here are the first few lines of the cleaned dataset of internet usage:
For this first dataset, we reduced the size from 6,570 observations across 4 variables to 3,433 observations for 4 variables.
Dataset on human freedom index
After importing the data from the CATO Institute website, we noticed that even if the file was called “Human Freedom Index 2022”, the available observations were only going from 2000 up to 2020. We have decided first to modify it in order to match our other datasets, by renaming/encoding/standardizing the columns containing the country names.
#### Human Freedom Index pre-cleaning 1 ####
data <- read.csv(here("scripts", "data", "human-freedom-index-2022.csv"))
#data in tibble
datatibble <- tibble(data)
# Rename the column countries into country to match the other datbases
names(datatibble)[names(datatibble) == "countries"] <- "country"
# Make sure the encoding of the country names are UTF-8
datatibble$country <- iconv(datatibble$country, to = "UTF-8", sub = "byte")
# standardize country names
datatibble <- datatibble %>%
mutate(country = countrycode(country, "country.name", "country.name"))Once done, we could verify which countries were or were not present between these observations and our main SDG dataset. We have decided to keep the ones that were matching between the two datasets.
#### Human Freedom Index pre-cleaning 2 ####
# Merge by country name
datatibble <- datatibble %>%
left_join(D1_0_SDG_country_list, by = "country")
datatibble <- datatibble %>% filter(code %in% list_country)
(length(unique(datatibble$code)))
#> [1] 159
# See which ones are missing
list_country_free <- c(unique(datatibble$code))
setdiff(list_country, list_country_free)
#> [1] "AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB"
# Turkey was missing but present in the initial database (it was a problem
# when standardizing the country names of D1_0SDG_country_list
#that we corrected) and the other missing countries are:
#"AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB"
D5_0_Human_freedom_index <- datatibbleThen, we noticed that there were a lot of columns that were not important for us, as we had 141 variables taken into account. So we have decided to keep the ones that refers to the countries informations (such as code, year, ..) and their human freedom scores per category (pf for personnal freedom, ef for economical freedom).
#### Human Freedom Index pre-cleaning 3 ####
# Erasing useless columns to keep only the general ones.
D5_0_Human_freedom_index <- select(D5_0_Human_freedom_index, year, country,
region, hf_score, pf_rol, pf_ss,
pf_movement, pf_religion, pf_assembly,
pf_expression, pf_identity, pf_score,
ef_government, ef_legal, ef_money, ef_trade,
ef_regulation, ef_score, code)
D5_0_Human_freedom_index <- D5_0_Human_freedom_index %>%
rename(
pf_law = names(D5_0_Human_freedom_index)[5], # Renames the 5th column to "pf_law"
pf_security = names(D5_0_Human_freedom_index)[6] # Renames the 6th column to "pf_security"
)Here are the first few lines of the partialy cleaned dataset on Human Freedom Index scores:
For this first dataset, we reduced the size from 3’465 observations across 141 variables to 3339 observations for 4 variables.
Dataset on Disasters
For this dataset concerning the Disasters we imported the data from Kaggle as we couldn’t find the original dataset that is private coming from the EOSDIS SYSTEM, an interactive interface for browsing full-resolution, global, daily satellite images from NASA. Once we made sure that our file called “Disasters” was convert into a data frame, we selected some specific columns that we where interested in.
#### Disasters pre-cleaning 1 ####
Disasters <- read.csv(here("scripts", "data", "Disasters.csv")) %>%
select(Year, Country, ISO, Location, Continent, Disaster.Subgroup,
Disaster.Type, Total.Deaths, No.Injured, No.Affected, No.Homeless,
Total.Affected, Total.Damages...000.US..)Because we knew that our file showed all the disasters in each country over the years (between 1970-2021) and that we wanted to focus on a specific period, we filtered our data to show the years between 2000 and 2022. Then we rearranged our data, changing the data types of all the columns and their names in order to match our other datasets.
#### Disasters pre-cleaning 2 ####
# Rearrange the columns, changed the type of data, renamed the columns
Rearanged_Disasters <- Disasters %>%
filter(Year >= 2000 & Year <= 2022) %>%
mutate(
code = as.character(ISO),
country = as.character(Country),
year = as.integer(Year),
continent = as.character(Continent),
disaster.subgroup = as.character(Disaster.Subgroup),
disaster.type = as.character(Disaster.Type),
location = as.character(Location),
total.deaths = as.numeric(Total.Deaths),
no.injured = as.numeric(No.Injured),
no.affected = as.numeric(No.Affected),
no.homeless = as.numeric(No.Homeless),
total.affected = as.numeric(Total.Affected),
total.damages = as.numeric(Total.Damages...000.US..)
)We then grouped the data by “year”, “code”, “country” and “continent” and summarize the data. Here you can see that we re-selected specific columns as we saw that our first pre-selection was still too wide and some variables as the disaster.subgroup and disaster.type weren’t pertinent.We arranged the columns based on “code,” “country,” “year,” and “continent” to match the other datasets.
#### Disasters pre-cleaning 3 ####
Disasters <- Rearanged_Disasters %>%
group_by(year,code, country, continent) %>%
summarize(
total_deaths = sum(total.deaths, na.rm = TRUE),
no_injured = sum(no.injured, na.rm = TRUE),
no_affected = sum(no.affected, na.rm = TRUE),
no_homeless = sum(no.homeless, na.rm = TRUE),
total_affected = sum(total.affected, na.rm = TRUE),
total_damages = sum(total.damages, na.rm = TRUE)
)
D6_0_Disasters <- Disasters %>%
select(code, country, year, continent, total_deaths, no_injured, no_affected,
no_homeless, total_affected, total_damages) %>%
arrange(code, country, year, continent)Finally we filtered our disasters data to keep only the countries that are present in our main dataset. We analysed the missing countries and identified three countries (BHR, BRN, MLT) that are unexpectedly missing.
#### Disasters pre-cleaning 4 ####
D6_0_Disasters <- D6_0_Disasters %>% filter(code %in% list_country)
length(unique(D6_0_Disasters$code))
#> [1] 163
# Here we see which countries are missing
list_country_disasters <- c(unique(D6_0_Disasters$code))
setdiff(list_country, list_country_disasters)
#> [1] "BHR" "BRN" "MLT"Here are the first few lines of the cleaned dataset on Disasters:
Dataset on COVID
This dataset contains information on the COVID19 pandemic between 2020 and 2022. The observation are by year, month, day. After importing the database, we transform the date in format YYYY-MM-DD in order to only keep the year.
#### COVID pre-cleaning 1 ####
COVID <- read.csv(here("scripts", "data", "COVID.csv")) %>%
select(iso_code, location, date, new_cases_per_million,
new_deaths_per_million, stringency_index) %>%
mutate(date = as.integer(year(date)))We perform a first round of investigation of the missing values before aggregating the values by year. We begin with the variables “cases per million” and “deaths per million”: seeing that for each country, we have either only missing values, either a very low percentage of missing values (~1%), we can compute the sum over each year and ignore the missing values without altering the data. Indeed, where all the values are missing, the computation will return a NA. We then look at the “stringency” variable and we have 3 scenarios:
~20% of missing values: we ignore missing values when computing the mean to have an idea of stringency each year (because we compute the mean stringency over the year, if some days are missing, it is not a problem, it can not evoluate that fast).
all are missing: we can ignore the missing values when computing the mean, because it will still return a missing value
almost all are missing: here the mean doesn’t make sense -> we will replace the values by NAs to be coherent. The countries with this issues are: ERI, GUM, PRI and VIR. We verify if they are in our main dataset and since none of these countries are, we can ignore the issue, the lines will be remove later anyway.
We aggregate the observations of all days of a year in one observation per country using the mean.
#### COVID missing values ####
COVID1 <- COVID %>%
group_by(iso_code) %>%
summarize(NaDeaths = round(mean(is.na(new_deaths_per_million)),2),
NaCases = round(mean(is.na(new_cases_per_million)), 2),
NaStringency = round(mean(is.na(stringency_index)), 2)) %>%
pivot_longer(cols = starts_with("Na"),
names_to = "Variable",
values_to = "NaValue")%>%
filter(NaValue!=0)
ggplot(COVID1,
aes(x = as.factor(NaValue),
fill = Variable)) +
geom_bar(stat = "count",
position = position_dodge2(preserve = "single"),
width = 0.35) +
labs(title = "Patterns of NAs for COVID variables before cleaning",
x = "proportion of NAs",
y = "Count of countries") +
scale_fill_viridis_d(name = "Variables",
na.value = "white",
begin = 0.5,
end = 1) +
theme_minimal() +
guides(fill = FALSE)
issue_list <- c("ERI",
"GUM",
"PRI",
"VIR")
is.element(issue_list, list_country)
#> [1] FALSE FALSE FALSE FALSE
COVID <- COVID %>%
group_by(location, date) %>%
mutate(
cases_per_million = sum(new_cases_per_million, na.rm = TRUE),
deaths_per_million = sum(new_deaths_per_million, na.rm = TRUE),
stringency = mean(stringency_index, na.rm = TRUE)
)%>%
ungroup()
Now that all the variables of interest are aggregated by year, we remove all the variables that we don’t need and rename all the remaining variables to match the main dataset.
#### COVID renaming ####
COVID <- COVID %>%
group_by(location, date) %>%
distinct(date, .keep_all = TRUE) %>%
ungroup()
COVID <- COVID %>%
select(-c(new_cases_per_million, new_deaths_per_million, stringency_index))
colnames(COVID) <- c("code",
"country",
"year",
"cases_per_million",
"deaths_per_million",
"stringency")We remove the years that exceed 2022, we make sure that the country codes are all iso codes with 3 letters (we observe that sometimes they are preceded by “OWID_”) and we standardize the country codes.
#### COVID years and code cleaning ####
COVID <- COVID[COVID$year <= 2022, ]
COVID$code <- gsub("OWID_", "", COVID$code)
COVID$code <- countrycode(
sourcevar = COVID$code,
origin = "iso3c",
destination = "iso3c"
)We remove the observations of countries that aren’t in our main dataset on SDGs and find that all the 166 countries that we have in the main SDG dataset are also in this one.
#### COVID pre-cleaned dataset ####
D7_0_COVID <- COVID %>%
filter(code %in% list_country)
length(unique(COVID$code))
#> [1] 238Here are the first few lines of the cleaned dataset on COVID19:
Dataset on Conflicts
For our conflicts dataset, we imported the data from “The World Banck” data catalog. Once we made sure that our file called “Disasters” was convert into a data frame, we selected some specific columns that we where interested in.
#### Conflicts dataset ####
Conflicts <- read.csv(here("scripts", "data", "Conflicts.csv")) %>%
as.data.frame() %>%
select(year, country, ongoing, gwsum_bestdeaths, pop_affected,
peaceyearshigh, area_affected, maxintensity, maxcumulativeintensity)Our file showed all the Conflicts and consequences per country over the years (between 2000-2016). We couldn’t find a better and more complete dataset, As we consider conflicts as events, we will only take into account results between 2000 and 2016. Then we rearranged our data, changing the data types of all the columns and their names in order to match our other datasets. We grouped the data by ” year”, “country”, re-selected some variables and summarize the data.
#### Conflicts rearranging 1 ####
Rearanged_Conflicts <- Conflicts %>%
filter(year >= 2000 & year <= 2022)%>%
mutate(
ongoing = as.integer(ongoing),
country = as.character(country),
year = as.integer(year),
gwsum_bestdeaths = as.numeric(gwsum_bestdeaths),
pop_affected = as.numeric(pop_affected),
area_affected = as.numeric(area_affected),
maxintensity = as.numeric(maxintensity),
)
# Group the data by "year", "country" and summarize the data
Conflicts <- Rearanged_Conflicts %>%
group_by(year, country) %>%
summarize(
ongoing = sum (ongoing, na.rm = TRUE),
sum_deaths = sum(gwsum_bestdeaths, na.rm = TRUE),
pop_affected = sum(pop_affected, na.rm = TRUE),
area_affected = sum(area_affected, na.rm = TRUE),
maxintensity = sum(maxintensity, na.rm = TRUE),
)After we Selected specific columns from the summarized data and arrange the data by our specified columns. To make our dataset compatible with the main one and let the merging face succeed, we dd some adjustment concerning the country names’ to ensure the compatibility. Then we standardize and merge by country names to finally rearrange the data to retain only the countries present in our main dataset. Note that in the end we can see that only one country is missing that wasn’t in the initial conflicts database: BLR
#### Conflicts rearranging 2 ####
conflicts <- Conflicts %>%
select(country, year, ongoing, sum_deaths,
pop_affected, area_affected, maxintensity) %>%
arrange(country, year)
conflicts$country <- iconv(conflicts$country, to = "UTF-8", sub = "byte")
conflicts <- conflicts %>%
mutate(country = countrycode(country, "country.name", "country.name"))
conflicts <- conflicts %>%
left_join(D1_0_SDG_country_list, by = "country")
conflicts <- conflicts %>%
select(code, country, year, ongoing, sum_deaths,
pop_affected, area_affected, maxintensity) %>%
arrange(code, country, year)
D8_0_Conflicts <- conflicts %>%
filter(code %in% list_country)
(length(unique(conflicts$code)))
#> [1] 166
# See which countries are missing
list_country_conflicts <- c(unique(conflicts$code))
setdiff(list_country, list_country_conflicts)
#> [1] "BLR"Here are the first few lines of the cleaned dataset on Conflicts:
Merging our dataset
By merging our eight pre-cleaned datasets, we create a final database.
#### Pre-cleaned datasets merged ####
D2_1_Unemployment_rate$country <- NULL
merge_1_2 <- D1_0_SDG |> left_join(D2_1_Unemployment_rate,
join_by(code, year))
D3_1_GDP_per_capita$country <- NULL
merge_12_3 <- merge_1_2 |> left_join(D3_1_GDP_per_capita,
join_by(code, year))
D3_2_Military_Expenditure_Percent_GDP$country <- NULL
merge_12_3 <- merge_12_3 |> left_join(D3_2_Military_Expenditure_Percent_GDP,
join_by(code, year))
D3_3_Miliraty_Expenditure_Percent_Gov_Exp$country <- NULL
merge_12_3 <- merge_12_3 |> left_join(D3_3_Miliraty_Expenditure_Percent_Gov_Exp,
join_by(code, year))
D4_0_Internet_usage$country <- NULL
merge_123_4 <- merge_12_3 |> left_join(D4_0_Internet_usage,
join_by(code, year))
D5_0_Human_freedom_index$country <- NULL
merge_1234_5 <- merge_123_4 |> left_join(D5_0_Human_freedom_index,
join_by(code, year))
D6_0_Disasters$country <- NULL
merge_12345_6 <- merge_1234_5 |> left_join(D6_0_Disasters,
join_by(code, year))
D7_0_COVID$country <- NULL
D7_0_COVID <- D7_0_COVID |> distinct(code, year, .keep_all = TRUE)
merge_123456_7 <- merge_12345_6 |> left_join(D7_0_COVID,
join_by(code, year))
D8_0_Conflicts$country <- NULL
all_Merge <- merge_123456_7 |> left_join(D8_0_Conflicts,
join_by(code, year)) Cleaning of the final database
Filing missing continents and regions colomns
When we merged our dataset, we noticed that some countries were not assigned their corresponding continents and/or region. This issue arose because we sourced the continent and region data from secondary databases, not from our main one. We now add this the corresponding missing continents and regions.
#### Filling missing continents and regions ####
# Update all_Merge with region and continent information
all_Merge <- all_Merge %>%
group_by(country) %>%
mutate(
continent = ifelse(is.na(continent), first(na.omit(continent)), continent),
region = ifelse(is.na(region), first(na.omit(region)), region)
) %>%
ungroup() %>%
mutate(continent = case_when(
code %in% c("BHR") ~ "Asia",
code %in% c("BRN") ~ "Asia",
code %in% c("MLT") ~ "Europe",
TRUE ~ continent
),
region = case_when(
code %in% c("AFG", "MDV") ~ "South Asia",
code %in% c("CUB") ~ "Latin America & the Caribbean",
code %in% c("STP", "SSD") ~ "Sub-Saharan Africa",
code %in% c("TKM", "UZB") ~ "Caucasus & Central Asia",
TRUE ~ region))We order the database, beginning by the information on the country, the year, the continent and the region.
#### Ordering the database and saving it as .CSV ####
all_Merge <- as.data.frame(all_Merge) %>%
select(code, year, country, continent, region, everything())
write.csv(all_Merge, file = here("scripts","data","all_Merge.csv"))Here are the first few lines of the final dataset:
Final structure of our merged database: each country of the 166 countries from D1_1_SDG are observed each year from 2000 to 2022, thus each row has a key composed of (code, year) that uniquely identifies an observation. The other columns are the variables listed above. Due to some countries having a lot of missing information we will have to eliminate some of them, but we will still have more than 2000 rows in our database.
Treatment of missing values
We load our final database and we visualize the missing values.We see that some variables have many NAs and that some patterns regarding row missingness emerge.
#### Loading the final database to be cleaned ####
# Import the final database
all_Merge <- read.csv(here("scripts","data","all_Merge.csv"))
# Remove unnecessary column
all_Merge <- all_Merge %>%
select(-c(X))
# Create a dataframe with the goals without NAs summarize in one column to
# simplify the visualization
goal_vars <- all_Merge %>%
select(starts_with("goal")) %>%
filter_all(all_vars(!is.na(.))) %>%
colnames()
to_plot_missing <- all_Merge %>%
mutate(Goals_without_NAs = rowSums(!is.na(select(., all_of(goal_vars))))) %>%
select(-c(goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9,
goal11, goal12, goal13, goal15, goal16, goal17))
vis_dat(to_plot_missing, warn_large_data = FALSE) +
scale_fill_viridis_d(na.value = "grey99",
begin = 0.4,
end = 0.9) +
theme(
axis.text.x = element_text(angle = 90, size = 6),
legend.text = element_text(size = 8), # Adjust the size of legend text
legend.title = element_text(size = 10)
)
For each of our research question, we will start with the merged data set and deal with the missing value separately, because there are often NAs for the same row across many columns that will be used for the same question. This will allow us to not delete observations when we do not need to.
For question 1, we only keep the years until 2020, because most of the explanatory variables that we want to use (those coming from the human freedom index) only have values until 2020.
#### Cleaning the database for question 1 ####
data_question1 <- all_Merge %>%
filter(year<=2020) %>%
select(-c(total_deaths, no_injured, no_affected, no_homeless, total_affected,
total_damages, cases_per_million, deaths_per_million, stringency,
ongoing, sum_deaths, pop_affected, area_affected, maxintensity))For question 2 and 4, we use the main data from the SDG database.
#### Cleaning the database for question 2 and 4 ####
data_question24 <- all_Merge %>%
select(c(code, year, country, continent, region, overallscore, goal1, goal2,
goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11,
goal12, goal13, goal15, goal16, goal17))For question 3, we create 3 distinct databases according to the different type of event that we will analyse: disasters, COVID19 and conflicts. For the disasters, we only keep the years until 2021, because after this date, we don’t have data, moreover we decided to delete the country Bahrain, Brunei and Malta as we do not have any data concerning them. For the conflicts, we only keep the years until 2016, because after this date, we don’t have data. Concerning the conflict dataset, we decided to erase Belarus because once again we do not have any data concerning this country.
#### Cleaning the database for question 3 ####
# Disasters
data_question3_1 <- all_Merge %>%
filter(year<=2021 & code!="BHR" & code!="BRN" & code!="MLT") %>%
select(c(code, year, country, continent, region, overallscore, goal1, goal2,
goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11,
goal12, goal13, goal15, goal16, goal7, total_deaths, no_injured,
no_affected, no_homeless, total_affected, total_damages))
# COVID
data_question3_2 <- all_Merge %>%
select(c(code, year, country, continent, region, overallscore, goal1, goal2,
goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11,
goal12, goal13, goal15, goal16, goal7, cases_per_million,
deaths_per_million, stringency))
# Conflicts
data_question3_3 <- all_Merge %>%
filter(year<=2016 & code !="BLR") %>%
select(c(code, year, country, continent, region, overallscore, goal1, goal2,
goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11,
goal12, goal13, goal15, goal16, goal7, ongoing, sum_deaths,
pop_affected, area_affected, maxintensity))Data for question 1
Dealing with missing values in colomns
We begin by visualizing the missing values. To have a less messy graph we group all the goals wihtout NAs into one single variable. We decide to remove MilitaryExpenditurePercentGovExp, because it has too many missing values and it contains similar information to MilitaryExpenditurePercentGDP.We also remove hf_score, pf_score and ef_score, because there are many missing values and since these variables summarize the other ones, deleting them will not make us loose information.
#### Visualizing missing values by variables ####
# Create a dataframe with the goals without NAs summarize in one column to simplify the visualization
variable_names <- names(data_question1)
missing_percentages <-
sapply(data_question1, function(col) mean(is.na(col)) * 100)
missing_data_summary <- data.frame(
Variable = variable_names,
Missing_Percentage = missing_percentages
)
missing_data_summary <- missing_data_summary %>%
mutate(VariableGroup = ifelse(startsWith(Variable, "goal") & Missing_Percentage == 0, "Goals without NAs", as.character(Variable)))
# Add a column with the number of missing values for the hover function
missing_data_summary$hover_text <- paste("Variable: ",
missing_data_summary$Variable,
"\nMissing percentage: ",
round(missing_data_summary$Missing_Percentage, 2),
"%", sep = "")
p <- ggplot(data = missing_data_summary,
aes(x = reorder(VariableGroup, Missing_Percentage),
y = Missing_Percentage,
fill = Missing_Percentage,
text = hover_text)) +
geom_bar(stat = "identity") +
scale_fill_viridis_c(name = "% NAs") +
labs(title = "Percentage of Missing Values by Variable",
x = "Variable",
y = "Missing Percentage") +
theme_minimal() +
theme(axis.text.y = element_text(hjust = 1,
size=6),
axis.text.x = element_text(angle = 45,
hjust = 1,
size=6),
legend.text = element_text(size = 8),
legend.title = element_text(size = 10)) +
guides(fill = FALSE)
# Convert to plotly and specify what to display when hovering the graph
ggplotly(p,
tooltip = "text") %>%
config(plot_ly,
displayModeBar = FALSE)
data_question1 <- data_question1 %>%
select(-c(MiliratyExpenditurePercentGovExp, hf_score, pf_score, ef_score))Dealing with missing values in rows
We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. We decide to remove the countries that have more than 50 missing values.
#### Colums with number of missing values ####
see_missing1_1 <- data_question1 %>%
group_by(code) %>%
summarise(across(-c(year, country, continent, region, population,
overallscore, goal1, goal2, goal3, goal4, goal5, goal6,
goal7, goal8, goal9, goal10, goal11, goal12, goal13,
goal15, goal16, goal17),
~ sum(is.na(.))) %>%
mutate(num_missing = rowSums(across(everything()))) %>%
filter(num_missing > 50))
data_question1 <- data_question1 %>% filter(!code %in% see_missing1_1$code)Here is the graph that allows us to visualize the countries that have missing values and how many , when there are more than 50 NAs in total.
#### Number of missing values per country (>50 NAs) ####
# Add a column with the number of missing values for the hover function
see_missing1_1$hover_text <- paste("Country: ", see_missing1_1$code,
"\nMissing values: ", see_missing1_1$num_missing)
# Creation of the plot with the number of missing values per country
p <- ggplot(see_missing1_1, aes(x = num_missing,
y = reorder(code, num_missing),
fill = num_missing,
text = hover_text)) +
geom_bar(stat = "identity") +
scale_fill_viridis_c(name = "Number of\nMissing Values",
begin = 0,
end = 1) +
theme_minimal() +
theme(axis.text.y = element_text(hjust = 1, size=8),
legend.text = element_text(size = 8),
legend.title = element_text(size = 10),
plot.title = element_text(size=12)) +
labs(title = "Number of missing values per country containing at least 50 NAs",
x = "Number of Missing Values",
y = "Countries") +
guides(fill = FALSE)
# Convert to plotly and specify what to display when hovering the graph
ggplotly(p, tooltip = "text") %>%
config(plot_ly, displayModeBar = FALSE)We also look at patterns of missing values in the rows and see that except for the two goals with NAs that we discussed earlier and for the triplet “ef_money”, “ef_trade” and “ef_regulation” there are not well defined patterns. We removes the countries that have NAs in the three variables mentioned at the same time.
#### Visualizing the missing values in the rows ####
gg_miss_upset(data_question1,
nsets=10,
nintersects=11)
data_question1 <-
data_question1[rowSums(is.na(data_question1[, c("ef_money",
"ef_trade",
"ef_regulation")])) < 3, ]
data_question1 <- data_question1 %>%
group_by(code) %>%
filter(all(2000:2020 %in% year)) %>%
ungroup()
GDP per capita
Only Venezuela has missing values that we can not fill (because the evolution over time is not linear), so we delete the country.
#### Deletion of Venezuela ####
question1_missing_GDP <- data_question1 %>%
group_by(code) %>%
summarize(NaGDPpercapita = mean(is.na(GDPpercapita)))%>%
filter(NaGDPpercapita != 0)
data_question1 <- data_question1 %>% filter(code!="VEN")Military expenditure in % of GDP
For MilitaryExpenditurePercentGDP, We plot the evolution of MilitaryExpenditurePercentGDP along the years for each country containing missing values and distinguish the percentage of missing values with colors.
#### Evolution of MilitaryExpenditurePercentGDP over the time ####
MilitaryExpenditurePercentGDP1 <- data_question1 %>%
group_by(code) %>%
summarize(NaMil1 = round(mean(is.na(MilitaryExpenditurePercentGDP)),3)) %>%
filter(NaMil1 != 0)
filtered_data_Mil1 <- MilitaryExpenditurePercentGDP %>%
filter(code %in% MilitaryExpenditurePercentGDP1$code) # countries with NAs
filtered_data_Mil1 <- filtered_data_Mil1 %>%
group_by(code) %>%
mutate(PercentageMissing = mean(is.na(MilitaryExpenditurePercentGDP))) %>% # Column % NAs
ungroup()
Evol_Missing_Mil1 <- ggplot(data = filtered_data_Mil1) +
geom_line(aes(x = year,
y = MilitaryExpenditurePercentGDP,
color = cut(PercentageMissing,
breaks = c(0,
0.1,
0.2,
0.3,
1),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")))) +
labs(title = "Military expenditure in % of GDP over time",
x = "Year",
y = "Military expenditure in % of GDP") +
scale_fill_manual(values = c("0-10%" = "#8dce3e",
"10-20%" = "#f7e424",
"20-30%"="#fba537",
"30-100%" = "#e34e64"),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")) +
guides(color = guide_legend(title = "% NAs")) +
facet_wrap(~ code, nrow = 5) +
theme(strip.text = element_text(size = 6),
axis.text.x = element_text(angle = 45, size= 6)) +
scale_y_continuous(breaks = NULL)
print(Evol_Missing_Mil1)
We delete the countries with more than 30% of missing values and for the countries with less than 30% of missing values and a linear evolution in time, we fill the missing values using linear interpolation.
#### Deletion of countries with (>30% NAs) ####
data_question1 <- data_question1 %>% filter(code!="ARE" &
code!="BHS" &
code!="BRB" &
code!="CRI" &
code!="HTI" &
code!="ISL" &
code!="PAN" &
code!="SYR" &
code!="VNM")
list_code <- c("BDI", "BEN", "CAF", "CIV", "COD",
"GAB", "NER", "TGO", "TTO", "ZMB")
for (i in list_code) {
country_data <- data_question1 %>%
filter(code == i)
interpolated_data <- na.interp(country_data$MilitaryExpenditurePercentGDP)
data_question1[data_question1$code == i, "MilitaryExpenditurePercentGDP"] <- interpolated_data
}Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the remaining missing values, where there are less than 30% missing using the median by region.
#### Distribution of the variable per region ####
question1_missing_Military <- data_question1 %>%
group_by(code) %>%
mutate(PercentageMissing = mean(is.na(MilitaryExpenditurePercentGDP))) %>% # Column % NAs
ungroup() %>%
group_by(region) %>%
filter(sum(PercentageMissing, na.rm = TRUE) > 0)
Freq_Missing_Military <- ggplot(data = question1_missing_Military) +
geom_histogram(aes(x = MilitaryExpenditurePercentGDP,
fill = cut(PercentageMissing,
breaks = c(0,
0.1,
0.2,
0.3,
1),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%"))),
bins = 30) +
labs(title = "Distribution of Military expenditures in % of GDP",
x = "Military expenditures in % of GDP",
y = "Frequency") +
scale_fill_manual(values = c("0-10%" = "blue",
"10-20%" = "green",
"20-30%"="red",
"30-100%" = "black"),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")) +
guides(fill = guide_legend(title = "% NAs")) +
facet_wrap(~ region, nrow = 1)
print(Freq_Missing_Military)
data_question1 <- data_question1 %>%
group_by(code) %>%
mutate(
PercentageMissingByCode = mean(is.na(MilitaryExpenditurePercentGDP))
) %>%
ungroup() %>%
group_by(region) %>%
mutate(
MedianByRegion = median(MilitaryExpenditurePercentGDP, na.rm = TRUE),
MilitaryExpenditurePercentGDP = ifelse(
PercentageMissingByCode < 0.3 & !is.na(MilitaryExpenditurePercentGDP),
MilitaryExpenditurePercentGDP,
ifelse(PercentageMissingByCode < 0.3, MedianByRegion, MilitaryExpenditurePercentGDP)
)
) %>%
select(-PercentageMissingByCode, -MedianByRegion)
Internet usage
There are only low percentage of missing values.
#### Percentage of missing values ####
question1_missing_Internet <- data_question1 %>%
group_by(code) %>%
summarize(NaInternet = mean(is.na(internet_usage)))%>%
filter(NaInternet != 0)There are never more than 30% of NAs. We look at the evolution of the variable over time. We fill the missing values with linear interpolation, because all are increasing in time and they are almost straight lines, except for CIV that we delete.
#### Evolution of the variable over time ####
question1_missing_Internet <- data_question1 %>%
group_by(code) %>%
mutate(PercentageMissing = mean(is.na(internet_usage))) %>% # Column % NAs
filter(code %in% question1_missing_Internet$code)
Evol_Missing_Internet <- ggplot(data = question1_missing_Internet) +
geom_line(aes(x = year,
y = internet_usage,
color = cut(PercentageMissing,
breaks = c(0,
0.1,
0.2,
0.3,
1),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")))) +
labs(title = "Evolution of internet usage over time",
x = "Year",
y = "Internet usage in %") +
scale_color_manual(values = c("0-10%" = "blue",
"10-20%" = "green",
"20-30%" = "red",
"30-100%" = "black"),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")) +
guides(color = guide_legend(title = "% NAs")) +
theme(axis.text.x = element_text(angle = 45, size= 6),
axis.text.y = element_text(size= 6))+
facet_wrap(~ code, nrow = 4)
print(Evol_Missing_Internet)
list_code <- setdiff(unique(question1_missing_Internet$code), "CIV")
for (i in list_code) {
country_data <- data_question1 %>%
filter(code == i)
interpolated_data <- na.interp(country_data$internet_usage)
data_question1[data_question1$code == i, "internet_usage"] <- interpolated_data
}
data_question1 <- data_question1 %>%
filter(code!="CIV")
Human freedom index
Personal freedom: law
The variable pf_law has (many) NAs, but only for one country: BLZ, so we decide to remove it.
#### pf_law has NAs only for BLZ ####
data_question1 <- data_question1 %>%
filter(code!="BLZ")Economic freedom: government
There are no more missing values, thanks to the previous steps.
Economic freedom: money
5 countries have missing values, but the percentage of missing values is always below 25%.
#### Missing values in 5 countries (<25%) ####
question1_missing_ef_money <- data_question1 %>%
group_by(code) %>%
summarize(Na_ef_money = mean(is.na(ef_money))) %>%
filter(Na_ef_money != 0)We look at the evolution of the variable over time, and for the countries with a linear evolution in time, we fill the missing values using linear interpolation.
#### Evolution of economic freedom: money over time ####
question1_missing_ef_money <- data_question1 %>%
group_by(code) %>%
mutate(PercentageMissing = mean(is.na(ef_money))) %>% # Column % NAs
filter(code %in% question1_missing_ef_money$code)
Evol_Missing_ef_money <- ggplot(data = question1_missing_ef_money) +
geom_line(aes(x = year,
y = ef_money,
color = cut(PercentageMissing,
breaks = c(0,
0.1,
0.2,
0.3,
1),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")))) +
labs(title = "Evolution of economic freedom: money over time",
x = "Year",
y = "ef_money") +
scale_color_manual(values = c("0-10%" = "blue",
"10-20%" = "green",
"20-30%" = "red",
"30-100%" = "black"),
labels = c("0-10%",
"10-20%",
"20-30%",
"50-100%")) +
guides(color = guide_legend(title = "% NAs")) +
theme(axis.text.x = element_text(angle = 45, size= 6))+
facet_wrap(~ code, nrow = 1) +
scale_y_continuous(limits = c(0, 10))
print(Evol_Missing_ef_money)
list_code <- c("GEO",
"MKD")
for (i in list_code) {
country_data <- data_question1 %>%
filter(code == i)
interpolated_data <- na.interp(country_data$ef_money)
data_question1[data_question1$code == i, "ef_money"] <- interpolated_data
}
Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the missing values using the median by region.
#### Evolution of economic freedom: money ####
question1_missing_ef_money <- data_question1 %>%
group_by(code) %>%
mutate(PercentageMissing = mean(is.na(ef_money))) %>% # Column % NAs
ungroup() %>%
group_by(region) %>%
filter(sum(PercentageMissing, na.rm = TRUE) > 0)
Freq_Missing_ef_money <- ggplot(data = question1_missing_ef_money) +
geom_histogram(aes(x = ef_money,
fill = cut(PercentageMissing,
breaks = c(0,
0.1,
0.2,
0.3,
1),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%"))),
bins = 30) +
labs(title = "Distribution of economic freedom: money",
x = "ef_money",
y = "Frequency") +
scale_fill_manual(values = c("0-10%" = "blue",
"10-20%" = "green",
"20-30%"="red",
"30-100%" = "black"),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")) +
guides(fill = guide_legend(title = "% NAs")) +
facet_wrap(~ region, nrow = 1)
print(Freq_Missing_ef_money)
data_question1 <- data_question1 %>%
group_by(code) %>%
mutate(
PercentageMissingByCode = mean(is.na(ef_money))
) %>%
ungroup() %>%
group_by(region) %>%
mutate(
MedianByRegion = median(ef_money, na.rm = TRUE),
ef_money = ifelse(
PercentageMissingByCode < 0.3 & !is.na(ef_money),
ef_money,
ifelse(PercentageMissingByCode < 0.3, MedianByRegion, ef_money)
)) %>%
select(-PercentageMissingByCode, -MedianByRegion)
Economic freedom: trade
6 countries have missing values, but the percentage of missing values is always below 25%.
#### Missing values in 6 countries (<25%) ####
question1_missing_ef_trade <- data_question1 %>%
group_by(code) %>%
summarize(Na_ef_trade = mean(is.na(ef_trade))) %>% # Column % NAs
filter(Na_ef_trade != 0)We look at the evolution of the variable over time. For the countries where this evolution is linear, we fill in the missing values using linear interpolation.
#### Evolution of economic freedom: trade over time ####
question1_missing_ef_trade <- data_question1 %>%
group_by(code) %>%
mutate(PercentageMissing = mean(is.na(ef_trade))) %>% # Column % NAs
filter(code %in% question1_missing_ef_trade$code)
Evol_Missing_ef_trade <- ggplot(data = question1_missing_ef_trade) +
geom_line(aes(x = year,
y = ef_trade,
color = cut(PercentageMissing,
breaks = c(0,
0.1,
0.2,
0.3,
1),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")))) +
labs(title = "Evolution of economic freedom: trade over time",
x = "Year",
y = "ef_trade") +
scale_color_manual(values = c("0-10%" = "blue",
"10-20%" = "green",
"20-30%" = "red",
"30-100%" = "black"),
labels = c("0-10%",
"10-20%",
"20-30%",
"50-100%")) +
guides(color = guide_legend(title = "% NAs")) +
theme(axis.text.x = element_text(angle = 45, size= 6))+
facet_wrap(~ code, nrow = 2) +
scale_y_continuous(limits = c(0, 10))
print(Evol_Missing_ef_trade)
# Linear interpolation for "AZE", "BFA", "ETH", "GEO", "VNH"
list_code <- c("AZE",
"GEO",
"MKD",
"MNG")
for (i in list_code) {
country_data <- data_question1 %>% filter(code == i)
interpolated_data <- na.interp(country_data$ef_trade)
data_question1[data_question1$code == i, "ef_trade"] <- interpolated_data
}
Then, we look at the distribution of the variable per region. Seeing that the only region that still has missing values is a centered distribution, we decide to replace the missing values using the mean of the region.
#### Distribution of ef_trade missing values ####
question1_missing_ef_trade <- data_question1 %>%
group_by(code) %>%
mutate(PercentageMissing = mean(is.na(ef_trade))) %>% # Column % NAs
ungroup() %>%
group_by(region) %>%
filter(sum(PercentageMissing, na.rm = TRUE) > 0)
Freq_Missing_ef_trade <- ggplot(data = question1_missing_ef_trade) +
geom_histogram(aes(x = ef_trade,
fill = cut(PercentageMissing,
breaks = c(0,
0.1,
0.2,
0.3,
1),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%"))),
bins = 30) +
labs(title = "Distribution of economic freedom: trade",
x = "ef_trade",
y = "Frequency") +
scale_fill_manual(values = c("0-10%" = "blue",
"10-20%" = "green",
"20-30%"="red",
"30-100%" = "black"),
labels = c("0-10%",
"10-20%",
"20-30%",
"30-100%")) +
guides(fill = guide_legend(title = "% NAs")) +
facet_wrap(~ region, nrow = 2)
print(Freq_Missing_ef_trade)
data_question1 <- data_question1 %>%
group_by(code) %>%
mutate(
PercentageMissingByCode = mean(is.na(ef_trade))
) %>%
ungroup() %>%
group_by(region) %>%
mutate(
MeanByRegion = mean(ef_trade, na.rm = TRUE),
ef_trade = ifelse(
PercentageMissingByCode < 0.3 & !is.na(ef_trade),
ef_trade,
ifelse(PercentageMissingByCode < 0.3, MeanByRegion, ef_trade)
)) %>%
select(-PercentageMissingByCode, -MeanByRegion)
Economic freedom: regulation
There are no more missing values, thanks to the previous steps.
SDGs 1 and 10
We noticed earlier that there were only missing values for goals 1 and 10. As we did before, we have started to investigate where are located the NAs in our dataset for first goal1, then goal 10.
#### Goal1 and Goal10 missing values ####
na_count <- sapply(data_question1, function(x) sum(is.na(x)))
na_count_df <- data.frame(variable = names(na_count),
num_missing = na_count)
na_count_df_filtered <- subset(na_count_df,
num_missing > 0)
ggplot(na_count_df_filtered,
aes(x= num_missing,
y=variable,
fill = num_missing)) +
geom_bar(aes(fill = num_missing),
stat = "identity",
width = 0.8,
fill = 'lightblue') +
geom_text(aes(label = num_missing),
vjust = 0.5,
hjust = 1.1,
position = position_dodge(width = 0.9)) +
theme_minimal() +
theme(axis.text.y = element_text(hjust = 1, size=10 ),
legend.text = element_text(size = 8),
legend.title = element_text(size = 10)) +
labs(title = "Number of remaining missing values per variable ",
x = "Number of NAs",
y = "Variables")
# goal1
question1_missing_goal1 <- data_question1 %>%
group_by(code) %>%
summarize(Na_goal1 = mean(is.na(goal1))) %>%
filter(Na_goal1 != 0)
data_question1 <- data_question1 %>%
filter(!code %in% question1_missing_goal1$code)
# still 42 NA values goal10
We had found that the missing values were located in only 5 countries. So we have decided to get rid of them. At this stage, there were only 42 remaining missing values. Then, we redo the same steps for goal 10.
#### Goal10 missing values ####
question1_missing_goal10 <- data_question1 %>%
group_by(code) %>%
summarize(Na_goal10 = mean(is.na(goal10))) %>%
filter(Na_goal10 != 0)
data_question1 <- data_question1 %>%
filter(!code %in% question1_missing_goal10$code)We have found the 2 lasts countries containing missing values. Now, our dataset is completely clean and ready to be used for our question 1.
Data for question 2 and 4
We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. Since there are no other missing values, we stop here.
#### Missing values columns ####
see_missing24 <- data_question24 %>%
group_by(code) %>%
summarise(across(everything(), ~ sum(is.na(.))) %>%
mutate(num_missing = rowSums(across(everything()))) %>%
filter(num_missing > 0))
#> `summarise()` has grouped output by 'code'. You can override using
#> the `.groups` argument.
data_question24 <- data_question24 %>%
group_by(country) %>%
filter(!all(is.na(goal1)) & !all(is.na(goal10)))Data for question 3
We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. Since there are no other missing values, we stop here.
Disasters
We begin by visualizing the missing values.
#### Visualizing missing values ####
variable_names <- names(data_question3_1)
missing_percentages <- sapply(data_question3_1, function(col) mean(is.na(col)) * 100)
missing_data_summary <- data.frame(
Variable = variable_names,
Missing_Percentage = missing_percentages
)
missing_data_summary <- missing_data_summary %>%
mutate(VariableGroup = ifelse(startsWith(Variable, "goal") & Missing_Percentage == 0, "Goals without NAs", as.character(Variable)))
ggplot(data = missing_data_summary, aes(x = reorder(VariableGroup, Missing_Percentage), y = Missing_Percentage, fill = Missing_Percentage)) +
geom_bar(stat = "identity") +
geom_text(aes(label = ifelse(Missing_Percentage > 1, sprintf("%.1f%%", Missing_Percentage), ""), y = Missing_Percentage),
position = position_stack(vjust = 1), # Adjust vertical position
color = "white", # Text color
size = 3, # Text size
hjust = 1.05) +
labs(title = "Percentage of Missing Values by Variable",
x = "Variable",
y = "Missing Percentage") +
theme_minimal() +
theme(axis.text.y = element_text(hjust = 1)) +
coord_flip()
In this particular case, even if there are many missing values in our disaster dataset, we made the hypothesis that disaster events can not happen every year for every country given that these are uncontrollable and non-recurring events. Therefore the NAs that we encounter will become zeroes, implying that there have been no climatic disasters.
#### Replacing NAs by 0 ####
data_question3_1[is.na(data_question3_1)] <- 0COVID19
We look at the missing values for the three variables that are specific to COVID during the years of COVID: 2020 to 2022. We delete the countries that have NAs (only stringency has 6 countries with 100% NAs).
#### COVID19 Missing values graphs ####
COVID4 <- data_question3_2 %>%
filter(year >= 2020 & year <= 2022) %>%
group_by(code) %>%
summarize(Na_deaths = mean(is.na(deaths_per_million)),
Na_cases = mean(is.na(cases_per_million)),
Na_stringency = mean(is.na(stringency))) %>%
filter(Na_deaths != 0 | Na_cases!=0 | Na_stringency !=0)
g1 <- ggplot(COVID4, aes(x = reorder(code, Na_deaths), y = Na_deaths)) +
geom_bar(stat = "identity",
fill = "lightgreen",
color = "black") +
labs(title = "NAs by country: \ndeaths per million",
x = "Country code",
y = "% NAs") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1, size=6),
plot.title=element_text(size=10),
axis.text.y = element_text(size= 6),
axis.title.x = element_text(size= 8),
axis.title.y = element_text(size= 8)) +
scale_y_continuous(limits = c(0, 1))
g2 <- ggplot(COVID4,
aes(x = reorder(code, Na_cases),
y = Na_cases)) +
geom_bar(stat = "identity",
fill = "lightgreen",
color = "black") +
labs(title = "NAs by country: \ncases per million",
x = "Country code",
y = "% NAs") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1, size=6),
plot.title=element_text(size=10),
axis.text.y = element_text(size= 6),
axis.title.x = element_text(size= 8),
axis.title.y = element_text(size= 8)) +
scale_y_continuous(limits = c(0, 1))
g3 <- ggplot(COVID4,
aes(x = reorder(code, Na_stringency),
y = Na_stringency)) +
geom_bar(stat = "identity",
fill = "lightgreen",
color = "black") +
labs(title = "NAs by country: \nstringency",
x = "Country code",
y = "% NAs") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1, size=6),
plot.title=element_text(size=10),
axis.text.y = element_text(size= 6),
axis.title.x = element_text(size= 8),
axis.title.y = element_text(size= 8))
g1 + g2 + g3
data_question3_2 <- data_question3_2 %>%
filter(!code %in% COVID4$code)
We replace the NAs of the other COVID columns (years 2000 t0 2019) by 0 (because we don’t have real missing, only introduced by merging with the other databases).
#### Replacing NAs by 0 ####
data_question3_2 <- data_question3_2 %>%
mutate(
cases_per_million = ifelse(is.na(cases_per_million), 0, cases_per_million),
deaths_per_million = ifelse(is.na(deaths_per_million), 0, deaths_per_million),
stringency = ifelse(is.na(stringency), 0, stringency)
)Conflicts
We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed.Two countries have missing values, we remove them (MNE and SRB).
#### Removing countries because of missing values ####
see_missing3_3 <- data_question3_3 %>%
group_by(code) %>%
summarise(across(-c(goal1, goal10), # Exclude columns "goal1" and "goal10"
~ sum(is.na(.))) %>%
mutate(num_missing = rowSums(across(everything()))) %>%
filter(num_missing > 0))
data_question3_3 <- data_question3_3 %>% filter(!code %in% c("MNE",
"SRB",
"SSD"))#### EXPORT as CSV ####
write.csv(data_question1, file = here("scripts","data","data_question1.csv"))
write.csv(data_question24, file = here("scripts","data","data_question24.csv"))
write.csv(data_question3_1, file = here("scripts","data","data_question3_1.csv"))
write.csv(data_question3_2, file = here("scripts","data","data_question3_2.csv"))
write.csv(data_question3_3, file = here("scripts","data","data_question3_3.csv"))